Deja Vu: Characterizing Worker Reliability Using Task Consistency
نویسندگان
چکیده
Consistency is a practical metric that evaluates an instrument’s reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker’s reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task — such as placement, difficulty, and transformation — affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.
منابع مشابه
Beyond Deja Vu in the Search for Cross-Situational Consistency
Recent efforts to resolve the debate regarding the consistency of social behavior are critically analyzed and reviewed in the light of new data. Even with reliable measures, based on multiple behavior observations aggregated over occasions, mean cross-situational consistency coefficients were of modest magnitude; in contrast, impressive temporal stability was found. Although aggregation of meas...
متن کاملDéjà vu experiences in healthy subjects are unrelated to laboratory tests of recollection and familiarity for word stimuli
Recent neuropsychological and neuroscientific research suggests that people who experience more déjà vu display characteristic patterns in normal recognition memory. We conducted a large individual differences study (n = 206) to test these predictions using recollection and familiarity parameters recovered from a standard memory task. Participants reported déjà vu frequency and a number of its ...
متن کاملAirspeed Calibration Using GPS
An Informal Memo by Wayne Olson .....................................................................1 Déjà vu – I’ve seen this before:.............................................................................
متن کاملThe renaissance of organic radical chemistry – deja vu all over again
Long-thriving movements often have a habit of continually reinventing themselves in a process called renaissance. The word itself is an English reinvention (some would say theft) of the French word for “rebirth”. Renaissances often occur in waves with each new wave typically building on the prior one. The European Renaissance is a perfect example, occurring in waves over about three centuries. ...
متن کامل